Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(backend): Add stricter URL validation for block requests #8890

Merged
merged 5 commits into from
Dec 6, 2024
Merged

fix(backend): Add stricter URL validation for block requests #8890

merged 5 commits into from
Dec 6, 2024

Conversation

majdyz
Copy link
Contributor

@majdyz majdyz commented Dec 5, 2024

We need stricter URL validation for the hostname we can request in the block code.

Changes 🏗️

  • Canonicalization: Ensures \ are converted to /, adds http:// if missing, and normalizes the input URL.
  • Scheme Check: Only http or https are allowed.
  • Hostname Validation:
    • Ensures a hostname exists.
    • Converts it to an IDNA ASCII form to prevent Unicode spoofing.
    • Verifies that the hostname matches a safe DNS pattern.
  • Trusted Origins Check: Allows certain hostnames explicitly if needed.
  • IP Resolution and Blocking:
    • Resolves the hostname to its IP addresses.
    • Checks against a list of private/reserved IP networks to prevent SSRF to internal services.

Checklist 📋

For code changes:

  • I have clearly listed my changes in the PR description
  • I have made a test plan
  • I have tested my changes according to the test plan:
    • ...
Example test plan
  • Create from scratch and execute an agent with at least 3 blocks
  • Import an agent from file upload, and confirm it executes correctly
  • Upload agent to marketplace
  • Import an agent from marketplace and confirm it executes correctly
  • Edit an agent from monitor, and confirm it executes correctly

For configuration changes:

  • .env.example is updated or already compatible with my changes
  • docker-compose.yml is updated or already compatible with my changes
  • I have included a list of my configuration changes in the PR description (under Changes)
Examples of configuration changes
  • Changing ports
  • Adding new services that need to communicate with each other
  • Secrets or environment variable changes
  • New or infrastructure changes such as databases

@majdyz majdyz requested a review from ntindle December 5, 2024 23:30
@majdyz majdyz requested a review from a team as a code owner December 5, 2024 23:30
@majdyz majdyz requested review from kcze and removed request for a team December 5, 2024 23:30
@github-actions github-actions bot added the platform/backend AutoGPT Platform - Back end label Dec 5, 2024
@netlify Netlify
Copy link

netlify bot commented Dec 5, 2024 •
edited
Loading

✅ Deploy Preview for auto-gpt-docs-dev canceled.

Name Link
🔨 Latest commit ff065cd
🔍 Latest deploy log https://app.netlify.com/sites/auto-gpt-docs-dev/deploys/675259c36486e2000825d307

@github-actions github-actions bot added the size/l label Dec 5, 2024
@qodo-merge-pro Qodo Merge Pro
Copy link

qodo-merge-pro bot commented Dec 5, 2024

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 Security concerns

DNS Rebinding:
The time-of-check to time-of-use gap between DNS resolution and actual request could allow an attacker to perform DNS rebinding attacks by changing DNS records after validation but before request execution. Consider implementing additional checks at request time or using a secure DNS resolver with DNSSEC validation.

⚡ Recommended focus areas for review

Possible Bug
The hostname validation regex HOSTNAME_REGEX may be too restrictive and could reject valid domain names that contain underscores in subdomains, which are technically valid according to DNS standards

Error Handling
The socket.getaddrinfo() call should handle more specific exceptions beyond just socket.gaierror, as it can raise other exceptions like socket.timeout

Performance Issue
DNS resolution for each URL validation could be slow and potentially vulnerable to DNS timing attacks. Consider implementing caching for resolved IPs with appropriate TTL

@netlify Netlify
Copy link

netlify bot commented Dec 5, 2024 •
edited
Loading

✅ Deploy Preview for auto-gpt-docs canceled.

Name Link
🔨 Latest commit ff065cd
🔍 Latest deploy log https://app.netlify.com/sites/auto-gpt-docs/deploys/675259c32b8f860008f2d938

@majdyz majdyz added this pull request to the merge queue Dec 6, 2024
Merged via the queue into dev with commit ffc3eff Dec 6, 2024
19 checks passed
@majdyz majdyz deleted the zamilmajdy/secrt-1010-ssrf-check-bypass-in-requests-utility-in-significant branch December 6, 2024 04:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ntindle ntindle ntindle approved these changes

@kcze kcze Awaiting requested review from kcze kcze is a code owner automatically assigned from Significant-Gravitas/maintainers

Assignees
No one assigned
Labels
platform/backend AutoGPT Platform - Back end Review effort [1-5]: 4 size/l
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@majdyz @ntindle